In a complex computing environment, computing, memory, and storage resources represent and can be organized into levels of aggregation. For example, a node represents the aggregate computing, memory, and storage resources of a physical server. A cluster of nodes represents the aggregate computing, memory, and storage resources of a group of physical servers. The cluster can act and be managed as a single entity. In managing the cluster, the nodes synchronize metadata for the aggregated resources. For example, a cluster may utilize a synchronized, distributed key/value system to represent the capabilities of storage resources of each node or other storage metadata.
In a distributed key/value system with a master-slave architecture, the master node synchronizes metadata by transmitting the metadata to a new node when the new node joins the cluster. In a partition-tolerant distributed key/value system, one or more nodes can be removed from the cluster, e.g., due to a network partition. The partitioned node(s) continue to function normally and independently using the key/value system. As a result, a partitioned node might have additional key/value entries that the cluster does not have and vice versa. After the network partition is resolved, the node that was partitioned rejoins the cluster. The act of rejoining the cluster creates a need for the master node to reconcile and synchronize key/value pairs between the rejoining nodes and the existing cluster of nodes. If performed solely by the master node, this reconciliation and synchronization can create a bottleneck for the system.